This disclosure relates generally to improving relevance of content presented to a user, such as a social networking system user, and more particularly to improving content relevance based on content ratings by a pool of human raters.
Certain online systems, such as social networking systems, allow users to connect to and to communicate with other users of the system. For a social networking system, for example, users create profiles on the social networking system that are tied to their identities and include information about the users, such as interests and demographic information. The social networking system selects and presents content to a user to encourage the user to interact with the social networking system and with other users of the social networking system. For example, the social networking system generates a feed of content items for presentation to a user that includes content items describing actions performed by other users of the social networking system or content provided to the social networking system by other users of the social networking system.
When selecting content for presentation to a user, social networking systems can determine measures of relevance between various content items and the user. The measure of relevance between a user and a content item is typically based at least in part on the user's likelihood of interacting with the content item when the content item is presented. The measure of relevance can often be determined using machine intelligence by, for example, applying an automated ranking system and machine learning model to determine the most relevant content for a user. However, in some cases, there can be factors that a human user might consider in a relevance determination that are not considered in a machine-based system.